Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding

A prominent challenge of offline reinforcement learning (RL) is the issue of hidden confounding: unobserved variables may influence both the actions taken by the agent and the observed outcomes. Hidden confounding can compromise the validity of any causal conclusion drawn from data and presents a major obstacle to effective offline RL. In the present paper, we tackle the problem of hidden confounding in the nonidentifiable setting. We propose a definition of uncertainty due to hidden confounding bias, termed delphic uncertainty, which uses variation over world models compatible with the observations, and differentiate it from the well-known epistemic and aleatoric uncertainties. We derive a practical method for estimating the three types of uncertainties, and construct a pessimistic offline RL algorithm to account for them. Our method does not assume identifiability of the unobserved confounders, and attempts to reduce the amount of confounding bias. We demonstrate through extensive experiments and ablations the efficacy of our approach on a sepsis management benchmark, as well as on electronic health records. Our results suggest that nonidentifiable hidden confounding bias can be mitigated to improve offline RL solutions in practice

Aerosol-jet-printed, conformable microfluidic force sensors.

Force sensors that are thin, low-cost, flexible, and compatible with commercial microelectronic chips are of great interest for use in biomedical sensing, precision surgery, and robotics. By leveraging a combination of microfluidics and capacitive sensing, we develop a thin, flexible force sensor that is conformable and robust. The sensor consists of a partially filled microfluidic channel made from a deformable material, with the channel overlaying a series of interdigitated electrodes coated with a thin, insulating polymer layer. When a force is applied to the microfluidic channel reservoir, the fluid is displaced along the channel over the electrodes, thus inducing a capacitance change proportional to the applied force. The microfluidic molds themselves are made of low-cost sacrificial materials deposited via aerosol-jet printing, which is also used to print the electrode layer. We envisage a large range of industrial and biomedical applications for this force sensor

Conformable and robust microfluidic force sensors to enable precision joint replacement surgery

Balancing forces within weight-bearing joints such as the hip during joint replacement is essential for implant longevity. Minimising implant failure and the corresponding need for expensive and difficult revision surgery is vital to both improve the quality of life of the patient and lighten the burden on overstretched healthcare systems. However, balancing forces during total hip replacements is currently subjective and entirely dependent on surgical skill, as there are no sensors currently on the market that are capable of providing quantitative force feedback within the small and complex geometry of the hip joint. Here, we solve this unmet clinical need by presenting a thin and conformable microfluidic force sensor, which is compatible with the standard surgical procedure. The sensors are fabricated via additive manufacturing, using a combination of 3D and aerosol-jet printing. We optimised the design using finite element modelling, then incorporated and calibrated our sensors in a 3D printed model hip implant. Using a bespoke testing rig, we demonstrated high sensitivity at typical forces experienced following implantation of hip replacements. We anticipate that these sensors will aid soft tissue balancing and implant positioning, thereby increasing the longevity of hip replacements. These sensors thus represent a powerful new surgical tool for a range of orthopaedic procedures where balancing forces is crucial

POETREE: Interpretable Policy Learning with Adaptive Decision Trees

Building models of human decision-making from observed behaviour is critical to better understand, diagnose and support real-world policies such as clinical care. As established policy learning approaches remain focused on imitation performance, they fall short of explaining the demonstrated decision-making process. Policy Extraction through decision Trees (POETREE) is a novel framework for interpretable policy learning, compatible with fully-offline and partially-observable clinical decision environments -- and builds probabilistic tree policies determining physician actions based on patients' observations and medical history. Fully-differentiable tree architectures are grown incrementally during optimization to adapt their complexity to the modelling task, and learn a representation of patient history through recurrence, resulting in decision tree policies that adapt over time with patient information. This policy learning method outperforms the state-of-the-art on real and synthetic medical datasets, both in terms of understanding, quantifying and evaluating observed behaviour as well as in accurately replicating it -- with potential to improve future decision support systems

Research data supporting "Aerosol-Jet Printed Conformable Microfluidic Force Sensors"

The repository includes original SEM image and data collections for the figures in the paper titled "Aerosol-Jet Printed Conformable Microfluidic Force Sensors" (Figure 3, 4, 5 and figures in Supplemental Information). The data was collected from experimental results and simulation results. The collection of experimental data includes means of impedance analyzer, Arduino board, force gauge, etc. The simulation method is based on Comsol Multiphysics

Research data supporting "Conformable and robust microfluidic force sensors to enable precision joint replacement surgery"

Research data supporting "Conformable and robust microfluidic force sensors to enable precision joint replacement surgery", to plot all figures in the main manuscript and supporting information. The data was collected from experiments and simulations. The experimental data was collected using an impedance analyser and load cell. Please see included README files for further detailsLI acknowledges support from an EPSRC Doctoral Training Partnership studentship (EP/R513180/1). TW acknowledges support from an EPSRC Doctoral Training Partnership studentship (EP/T517847/1). SKN is grateful for support from ERC Starting Grant (Grant No. ERC-2014-STG-639526, NANOGEN). SKN and QJ acknowledge support from the Centre of Advanced Materials for Integrated Energy Systems ''CAM-IES'' grant EP/P007767/1. JC was currently supported by a Wellcome Trust Institutional Strategic Support Award to the University of Exeter (204909/Z/16/Z); for the purpose of Open Access, the author has applied a CC BY public copyright license to any Author Accepted Manuscript version arising from this submission